Stability and robustness analysis of cooperation cycles driven by destructive agents in 

finite populations 
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The emergence and promotion of cooperation is one of the main issues in evolutionary game the- 
ory, as cooperation is amenable to exploitation by defectors, which take advantage from cooperative 
individuals at no cost dooming them to extinction. It has been recently shown that the existence of 
purely destructive agents (termed jokers) acting on the common enterprises (public goods games), 
can induce stable limit cycles between cooperation, defection and destruction when infinite popu- 
lations are considered. These cycles allow for time lapses in which cooperators represent a relevant 
fraction of the population, providing a mechanism for the emergence of cooperative states in nature 
and human societies. Here we study analytically and through agent-based simulations the dynamics 
generated by jokers in finite populations for several selection rules. Cycles appear in all cases stud- 
ied thus showing that the joker dynamics generically yields a robust cyclic behavior not restricted 
to infinite populations. We have also computed the average time in which the population consists 
mostly of just one strategy and compare the results with numerical simulations. 
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I. INTRODUCTION 

Cooperation is necessary for the appearance of com- 
plex structures and higher order selection units from in- 
dividual behaviors. In this way, cooperation between uni- 
cellular life forms gave rise to multicellular organisms, 
cooperative animals form communities and cooperation 
between humans gives rise to the complex societies we 
live in Thus, it is very important to understand the 
conditions that allow cooperation to thrive and evolve in 
nature and society. However, cooperative behaviors are 
not stable, as they are easily invaded by selfish individ- 
uals, who benefit from the interactions with cooperative 
ones but avoid paying the costs attached to cooperation. 
The selfish individuals, called defectors, have a higher 
fitness — a measure of reproductive success — than co- 
operators in any well mixed population, and therefore 
will spread under the action of natural selection, leading 
to the extinction of the cooperative behavior (H, and to 
populations where nobody benefits from altruistic acts 
(the "tragedy of the commons" 0]). 

In the last decades the study of the Public Goods (PG) 
game, a mathematical metaphor of a common enterprise, 
in which cooperative individuals invest — pay a cost — 
and share the benefits with all the players involved in 
the game, has led to the discovery of some mechanisms 
that allow cooperation to thrive, as introducing reputa- 
tion Q , diversity in number and size of groups , link- 
ing group size and payoffs @ , or the inclusion of spatial 
structure and conditional behaviors Furthermore, 
it has been proven that the introduction of some behav- 
ioral types in well- mixed population, as punishers [lol [ll| 



or individuals which do not participate in the PG and in- 
stead receive a fixed benefit (the so-called 'loners') [l^ . 
may promote cooperation. However, the only behavioral 
type found so far that allows for the emergence of sta- 
ble cycles in the presence of mutations is the so called 
joker strategy [l^. Jokers do not take part of the bene- 
fits produced by the PG game and instead they provoke 
a damage to the common enterprise, thus affecting every 
individual involved in it. Surprisingly, the effect of these 
indiscriminate destructive agents on the dynamics is the 
induction of robust evolutionary stable limit cycles of co- 
operation, defection and destruction. A cyclic dynamics 
can also be found when loners are involved. However, 
jokers induce cycles that are dynamically different from 
those found with loners [l^l, because the latter are neu- 
trally stable (i.e, have no fixed amplitudes) and disappear 
in the presence of mutations or structural noise. 

The inclusion of destructive agents in the PG game is 
motivated by the observation that, in nature and society, 
the appearance of a risk, as those created by common 
enemies, predators or simply dangerous situations, may 
induce cooperation among the victims [3, [l^ ■ 

In a previous work (Tsj , the stability of the evolution- 
ary cycles induced by jokers was proven for infinite pop- 
ulations through the analysis of the replicator mutator 
equation. Here, we extend the study to finite populations 
and analyze the dynamics for different updating methods 
p^ . i.e., different selection dynamics for the population, 
in order to analyze if robust cycles are also found. The 
conclusion we reach from this study is that the robust cy- 
cles obtained using the replicator-mutator equation are 
not just restricted to this particular dynamics but are a 
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generic feature of the joker modeL 

The paper is organized as follows. In section II we 
explain the PG game and review the dynamics in in- 
finite populations. Section III provides the stochastic 
equations describing the evolutionary dynamics for finite 
populations. In section IV we analyze the joker dynamics 
in finite populations using different selection dynamics in 
order to check the existence of cycles. Section V is de- 
voted to conclusions. 



II. EVOLUTIONARY CYCLES INDUCED BY 
JOKERS IN PUBLIC GOOD GAMES 

In each PG game a number n of individuals is randomly 
chosen from the entire population. Each cooperative in- 
dividual contributes to the common enterprise at a cost c 
to itself, which yields a benefit b = rc {r > 1) equally dis- 
tributed between all players; defectors free-ride the public 
good at no cost, thus obtaining a higher benefit than co- 
operators; jokers do not participate on the benefits, and 
each one provokes a damage —d < to be shared by all 
individuals engaged in the PG game. Note that the cost 
paid by C players can be set to c = 1 without loss of 
generality: all other payoffs are given in units of c. 

Let us call Q < m < n the number of cooperative 
participants, < < n the number of jokers, n~m~j > 
the number of defectors and S ~ n — j the number of 
non-jokers, i.e., the number of individuals involved in the 
PG game, and that potentially benefit from the PG. Then 
the payoff of a defector will be UY){m,j) = (rm — dj)/S, 
and that of a cooperator He = IId — 1; as previously 
stated, in each interacting group defectors will always do 
better than cooperators. Jokers' payoff is always 0. 

In summary, we have 



nD(m, j) 



rm — dj 
S ' 



0<m<S'-l<n-l, 



nc(m,j) = ^ - 1, l<m<S<n, (1) 



nj(m,j) =0, 



< S* < n - 1. 



A simple invasion analysis [l3| shows that the PG thus 
defined determines a tragedy of the commons Q for r < 
''max = n{AI — 1)/(M — n), where M is the population 
size. For infinite populations, the latter condition reduces 
to r < n, as usual in PG games. Therefore, a mixed 
population of cooperators and defectors with r < rmax 
will end up composed of defectors only. The interesting 
question is then if jokers may prevent the extinction of 
cooperators and under which conditions. In Ref. it 
was shown that, in the region of interest, namely 1 < r < 
^maxi d > 0, the system exhibits: (a) joker-cooperator 
bistability for l + d/{M -1) < r < l + {n-l)d, (b) joker 
dominance for r < 1 + d/{M — 1) and, most importantly, 
(c) a rock-paper-scissor (RPS) cyclic dominance of the 
three strategies for 



This condition expresses the fact that a single coopera- 
tor gets a positive payoff in spite of the damage inflicted 
by ri — 1 jokers, which allows cooperators to thrive in 
the damaging environment that represents a population 
of jokers and re-establish a cooperative environment. In 
Ref. [l3l |. we analyzed just one dynamics, namely the 
replicator-mutation dynamics, and showed that it pro- 
duces stable (robust) limit cycles C— ?>D— ^J^-C when mu- 
tations are rare, and stable coexistence for high mutation 
rates (Fig. [1]). In the following we will analyze the dy- 
namics of PG with jokers in finite populations under dif- 
ferent update rules, check the appearance of cycles and 
calculate the average time spent in each homogeneous 
state, in order to decide which dynamics better promotes 
the survival of cooperation. 



III. STOCHASTIC DYNAMICS IN FINITE 
POPULATIONS 

The deterministic evolution represented by the 
replicator-mutator equation is an idealization of the sys- 
tem behavior in the limit of infinite populations. To get 
a deeper insight into the model we need to address the 
question what happens when populations have a finite 
size A/. To begin with we need to describe the micro- 
scopic dynamics in more detail. Hauert et al. have 
proposed a protocol in which random selections of n play- 
ers are gathered together to play the game. After receiv- 
ing their corresponding payoffs the group dissolves and a 
new one is sampled. This sampling is made a sufficient 
number of times so that on average each player receives 
a payoff proportional to the mean payoff she can obtain 
given the composition of the population. 

Suppose there are m cooperators, j jokers, and M ~ 
m — j defectors in the population. The probability that 
the sampling of n individuals contains k cooperators, I 
jokers, and n — k — I defectors is given by the extended 
hypergeometric distribution 



p{k,l\n,m,j,M) 



J \ ( M — m — j 

n — k — I 



(3) 



r > 1 + {n- l)d. 



(2) 



The average payoff of strategy X within this population, 
Px{iTi, j), is obtained by averaging formulae ([!]) with this 
probability distribution. This is done in Appendix [XI 
where explicit expressions for Pc{m,j) and PY){m,j) are 
obtained — obviously Pj{m,i) = irrespective of the 
population composition. 

Once payoffs arc obtained evolution proceeds by im- 
itation. Different payoff-dependent updating rules have 
been proposed in the literature [l^. All of them de- 
scribe a process of birth and death which is defined by 
the transition probability Tim! , j'\m, j) from a popula- 
tion with composition {m,j) to another one with compo- 
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FIG. 1: (Color online.) Cycles induced by jokers in infinite populations. The simplexes describe the replicator-mutator 
dynamics for a population of cooperators, defectors and jokers with parameter values satisfying n > r > 1 + d{n — 1), for which 
a rock-paper-scissor dynamics is expected. For small mutation rates, the only equilibrium is a repeller (white dot in (a),(b)), 
and trajectories end up in a stable limit cycle of decreasing amplitude with increasing ^ (black line); when mutations reach a 
critical value fic, the system undergoes a Hopf bifurcation and a stable mixed equilibrium appears (black dot in (c)). Thus the 
presence of jokers induces periodic bursts of cooperation for low mutation rates, and stable coexistence for high /j.. Parameters: 
n = 5, r = 3, d = 0.4, /i is (a) 0.001, (b) 0.005, (c) 0.05. (Images generated using a modified version of the Dynamo Package 
0). 



sition (m',j') within the set 

Mnj = {{m,j), {m ± 1, j), {mj ± 1), 
(TO + l,j-l),(m-l,j + l)}. 



(4) 



If now Il{m,j;t) denotes the probability that the popu- 
lation has a composition given by {m,j) at time t, then 
this probability evolves according to 

n(m,j;t + l)= J2 Tim,j\m'j')n{m'j';t). 

(5) 

It is implicitly asumed that n(m, j; i) = for all t if the 
pair {m,j) is outside the set V = {{m,j) G : to, j > 0, 
m + j< M}. 

If we introduce matrix T, with elements T{m,j; m' 
[(m,j) is the "row index" and {m',j') the "column in- 
dex"] defined as 



T{m,j;m',j') 



T{m,j\m',f) if (to',/) eMn,,, 







otherwise. 



(6) 

and vectors n(t), with elements Il{m,j;t) [where 
{m,j), {m' e "P], then Eq. ([5]) can be cast in matrix 
notation simply as 



u{t + i) = Tn(i). 



A. Stationary state 



(7) 



If the process undergoes mutations then matrix T is 
ergodic and Eq. ^ has got a unique stationary state, tt. 



which is obtained by solving the linear system 



TT = Ttt. 



(8) 



In the absence of mutations, though, there are three ab- 
sorbing states corresponding to the three homogeneous 
populations. A homogeneous population remains invari- 
ant because the imitation process cannot change its com- 
position. We will denote these vectors ec, cd, ej, the 
index denoting the strategy of the homogeneous popula- 
tion. Clearly ec(TO,j) = (5,„,m'5j,o, en{m,j) = (5„i,o<5j,o, 



B. Infinitely small mutation rate 

After every imitation attempt (whether successful or 
not), individuals can randomly mutate their strategy. 
With probability 2/i the actor of the imitation event 
changes its current strategy into one of the other two 
equally likely. Parameter fi is referred to as the mutation 
ratio. In this section we will be concerned with mutation 
rates ^ 1. 

In the the limit — > 0+ the stationary probability dis- 
tribution must be a linear combination of the stationary 
vectors of the process without mutations, so in principle, 
taking the limit 



lim 



E 

=C,D,J 



ax ex 



(9) 



should provide the coefficients ax of this linear combi- 
nation, but this limit cannot be obtained directly from 
Eq. (jS]). There is an alternative though. It has been 



4 



proven [18| that the — 0+ hmit of this process is 
equivalent to another process with three states, C, D, 
J, in which the transition probability between X and Y 
is equal to the probability that a single mutant of type Y 
invades an otherwise homogeneous population of X indi- 
viduals, thus transforming it into a homogeneous popu- 
lation of Y individuals. Intuitively, this is tantamount to 
saying that mutations arc so rare that the ultimate fate 
of a mutant is decided before the next mutation occurs. 
The stationary vector in this space, 

a = {ac,aT),aj), (10) 

provides the values of the coefficients ax in 

Following [ll[, let pyx denote the probability that a 
single Y mutant takes over the population made of the 
mutant and M — 1 individuals of type X. Then the tran- 
sition probability of going from state X to a different 
state Y in the three-states Markov chain defined above 
will be ryx = PyxM- Introducing R = (ryx) so that the 
elements in each column add up to one (this fixes the 
diagonal of the matrix), we can rewrite this matrix as 
R = I + ^Q, where 

(-PDC - PJC PCD PC3 \ 

Pdc -PCD - Pjd Pdj • (11) 

P,JC Pjd -pcj - PD.I / 

Vector a. is then the solution of the linear system Qa = 
0. A little bit of algebra leads to the result 

ac = (pcDPcj + PCDPDj + pc.iPjd)M, (12) 
a-D = (PDCPDJ + /ODC/OCJ + PDj/3jc)M, (13) 
aj = (PJCPJD + AJCPCD + PJDPDC)M, (14) 

with A chosen so as to fulfill 

ax = l. (15) 

X=C,D,J 

C. Finite mutation rates 

If the mutation rate is not zero the Markov chain is 
ergodic and the stationary state can be obtained by solv- 
ing numerically Eq. ([8]) . This is accomplished with better 
accuracy by splitting 

T = To + Ti, (16) 
TT = ^ axex + TTi, (17) 

X=C,D,J 

with To the transition matrix in the absence of 
mutations — i.e., with transitions describing only the im- 
itation process. Then Tti is the solution of the linear 
system 

(I-T)7ri= ^ axTiex. (18) 

X=C,D,J 



D. Imitation rules 

In order to specify the transition matrix T we need 
to describe the imitation process. Of the many differ- 
ent rules applied in the literature [3 we have chosen 
the three most commonly employed: unconditional im- 
itation, proportional update, and a Moran process. In 
all cases the corresponding matrix T is obtained in Ap- 
pendix |B] 

Under unconditional imitation two players arc chosen 
at random among the population, one as the focal player 
and the other one as the model to imitate. The focal 
player compares both payoffs and changes her strategy 
to that of the model if the latter has a higher payoff. 
In this case, the strategy with the highest fitness never 
changes except by mutation, which is the only source of 
stochasticity in this rule. 

Appendix [Cl discusses the value ct for this update rule. 
There are two possibilities: 

(i) r > l + {n — l)d. In this cases all three homogeneous 
states are equally likely [c.f. Eq. (jC2l) ]. 

(ii) r < l + (n— l)(i. In this cases J is the only absorbing 
state of the process [c.f. Eq. (|C4[) ]. 

Proportional update is entirely similar to uncondi- 
tional imitation with the exception that imitation occurs 
with probability proportional to the payoff difference be- 
tween the model and the focal players. For this reason 
the values of a. for this rule are the same as those for 
unconditional imitation. 

In a Moran process a strategy is chosen to be imi- 
tated (or reproduced) with a probability proportional to 
its population-dependent fitness. The player who imi- 
tates (or is replaced by the offspring of) this selected 
player is randomly chosen from the rest of the popula- 
tion. The only drawback of this rule is that fitnesses 
must be positive for it to make sense, so they cannot be 
directly the payoffs of the game, because they can take 
negative values. A standard mapping between payoff and 
fitness is obtained by introducing the selection strength 
s [l^. This weights the contribution of the game to the 
total fitness of the strategy as = 1 — s + sP, with P 
the average payoff. Bounding the value of s we can force 
F to be positive. 

The Moran process thus described defines a birth- 
death process with two absorbing states, and the cor- 
responding probabilities pxY are obtained via standard 
formulae (see Appendix [C|) . 

IV. RESULTS: ROBUSTNESS OF THE CYCLES 
USING DIFFERENT SELECTION DYNAMICS 

In this section we compare the results of agent-based 
simulations with those obtained by solving the station- 
ary equation Simulations implement the following 
stochastic process. Wc start with a population of M 
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FIG. 2: (Color online.) Time evolution of the frequencies of the three strategies in a population of M = 1000 (left) and 
M — 100 (right) individuals playing a PG with jokers with different update rules: (a), (b) unconditional imitation, (c), (d) 
proportional update, and (e), (f) a Moran process. The presence of jokers induces a cyclic behavior irrespective of the update 
rule and the population size, as long as the mutation rate ^ > 0. Black solid lines: cooperators, red dashed lines: defectors, and 
blue dotted lines: jokers. One period corresponds to one updating event according to the evolutionary rule used. Parameters 
n = 5, r = 3, d = 0.4, fi = 0.001; in (a)-(d) s = 1, in (e),(f) s = 0.38. 



individuals with equal amounts of C, D and J players. 
Then: 

1. Assuming that every time step each individual 
plays many rounds of the game with different, ran- 
domly gathered groups of n players, the payoffs 
they obtain will be proportional to the average pay- 
offs, as calculated in Appendix El Thus we assume 
that these expressions provide the payoffs each in- 
dividual gains every time step. 

2. These payoffs are used to update the population 
according to the corresponding imitation rule. We 
implement the three rules described in Sec. IIII Dl 

3. With probability fi each newborn mutates to a dif- 
ferent strategy (any of the other two with equal 
probability). 

A quite general result is that, irrespective of the pop- 
ulation size, at low mutation rates simulations show 
patterns of cyclic invasions C— >D— >J— >-C (see Fig. 
These patterns resemble the limit cycles observed in 
the replicator dynamics, i.e., for infinite populations [fsj 
(c.f. Fig. HI). 

Roughly speaking we can distinguish three regimes of 
mutations. In the low mutation regime the system spends 
most of the time in homogeneous states, and the dy- 
namics of the system is well described by the ii ^ 
limit of the stationary probability distribution tt. This 
can be clearly seen in Fig. [31 The dashed-dotted curves 
in Figs. |3ja), (c) and (e) represent the fraction of time 
spent in transients when a homogeneous population is 
replaced by another one arisen as the result of muta- 
tions. This fraction is very small for fi < 10~^-10~"', 



depending on the imitation rule. For larger mutation 
rates (10~^-fO~'^ ^ M ^ 10~^-10~^) the system spends 
as much time in homogeneous populations as in mixed 
transient states. This is the regime displayed in Fig. [51 
where cycles are clearly defined even though for some 
imitation rules (particularly so for proportional update) 
certain homogeneous populations that are hardly ever 
reached [Fig.|5Jb) shows burst of cooperators which never 
reach a fraction higher than 80% of the population] . For 
even higher mutation rates homogeneous populations are 
very rare and the behavior of the system is very different, 
typically dominated by defectors [see FigsElJb), (d) and 
(f)]- 

Unconditional imitation is practically a deterministic 
rule in the low mutations regime. For fi < 10"^ the 
population is almost always homogeneous, and is made 
of each of the three strategies with equal probability [see 
FigS-HIa), (b) and Figs.jSlJa), (b)]. FigurejSlJa) shows this 
probability as a function of the joker's inflicted damage 
d. As long as d > and r > 1 + {n — l)d we find each 
strategy equally likely. For r<f-|-(n— l)da homo- 
geneous population of jokers cannot be invaded because 
this is the only absorbing state of the Markov chain for 
/i = 0. For d = jokers do not inflict damage. Then 
the system spends most of the time in a homogeneous 
population of defectors. However, random drift allows 
for occasional invasions by jokers, who are subsequently 
wiped out by cooperators, who in its turn get replaced 
again by defectors. Figure SJb) illustrates a typical real- 
ization exhibiting one of these turn-overs. 

As of proportional update, its main difference with 
unconditional imitation is its being a truly probabilis- 
tic rule, in which individuals only imitate higher payoffs 
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FIG. 3: (Color online.) Relative times spent in homogeneous as well as in transient states in a population of A'l = 100 
individuals. For practical purposes, a state is considered homogeneous if more than 95% of individuals belong to the same 
strategy. Symbols are the result of agent-based simulations; lines are obtained from the solution of Eqs. (|17|) - (|18p . Results 
for cooperators are represented with (black) squares and solid lines, those for defectors with (red) circles and dashed lines, 
and those for jokers with (blue) triangles and dotted lines. Panels (a), (c) and (e) also show (with inverted triangles and 
dashed-dotted lines) the fraction of time spent in transient states. Panels (b), (d) and (f) show the fractions of the time spent 
in each of the three homogeneous states relative to the total time spent in homogeneous states. Panels (a) and (b) correspond 
to unconditional imitation, panels (c) and (d) to proportional update, and panels (e) and (f) to a Moran rule. We can see 
that high mutation rates promote defection over the other two strategies. Parameters used are n = 5, r = 3, d = 0.4; selection 
strength is s = 1 in (a)-(d) and s — 0.38 in (e) and (f). 



with a certain probability. Although in the small muta- 
tions regime this leads to the same probability of mutual 
invasion of strategies as for unconditional imitation, the 
stochastic nature of this rule renders much longer inva- 
sion times. This can be clearly appreciated in Fig. [2] 

Another effect of stochasticity is that the time spent 
in transient states is also longer, thus shrinking the low 
mutations regime by more than one order of magnitude 
[compare Figs. [Sja) and (c)]. The effect is particularly 
notorious for jokers, who take a long time to invade de- 
fectors, thus extending the Hfe time of defective popula- 
tions. This effect is illustrated in Fig.[5l which represents 
a typical realization of an agent-based simulation. 

The Moran process is the randomest of the three evo- 
lutionary dynamics because even strategies not perform- 
ing very well have a chance to get imitated. The effect is 
more noticeable the smaller the population. This dynam- 
ics imposes an upper limit to the selection strength s (see 
Sec. lIIIDl) and the probabilities to find the population in 



each of the three homogeneous states depend on the pa- 
rameters of the game and on s in a nontrivial way (see 
Sec. lC 2[) . These probabilities are represented in Fig.[HJa) 
as a function of s. The theoretical predictions of Sec. lC 21 
agree with the simulations. This figure shows that coop- 
eration is highly promoted for small s(0.005 < s < 0.05). 
fn this limit cooperative populations are found with al- 
most 50% probability. This probability decreases down 
to around 25% for larger s. Figure [6Ub) shows a typical 
realization of this process, exhibiting a defining feature 
of this process, namely the frequent failures of attempted 
invasions. 

Whichever the update rule, when mutation rates are 
not small the system is better characterized by provid- 
ing the stationary probabihty distribution tt, as obtained 
from Eq. The results are plotted in Fig. [7] for aU 

three imitation rules and different mutation rates ^. For 
low and intermediate values of ^ the higher probabilities 
are found near the border of the simplexes, consistent 
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FIG. 4: (Color online.) Evolution of a population of M = 
100 individuals by unconditional imitation, (a) Fraction of 
time spent in homogeneous populations of cooperators (black 
squares and solid line) , defectors (red circles and dashed line) 
and jokers (blue triangles and dotted line), as a function of 
joker's inflicted damage d. Symbols correspond to an agent- 
based simulation; lines to the results obtained from numerical 
computation of the stationary probability distribution, (b) A 
realization made with d = showing an invasion of defectors 
by jokers through pure drift, and the subsequent burst of 
cooperators and turn-over by defectors. Parameters: n = 
5,r = 3, s = 1 and /i = 5 x 10"^. 



FIG. 6: (Color online.) A population of M = 100 individuals 
evolving through Moran update, (a) Comparison of the the 
relative times in which the population is in a homogeneous 
state vs. the selection strength, s, for low mutation rates. 
Lines represent the analytical estimates obtained in Sec. IC 21 
symbols represent the results from agent-based simulations, 
(b) Fractions of each strategy as a function of time as obtained 
from a realization of an agent-based simulation. Cooperators 
are represented with a black solid line and squares, defectors 
with a red dashed line and circles, and jokers with a blue 
dotted line and triangles. Parameters are n = 5,r = 3, d = 0.4 
and jj. — 5 X 10^^. In (b) the selection strength is s = 0.05. 



DISCUSSION AND CONCLUSIONS 
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FIG. 5: (Color online.) Realization of an agent-based sim- 
ulation of a population with M = 100 individuals evolving 
through proportional update. Notation is as in Fig. |4l Pa- 
rameters: n = 5, r = 3, d = 0.4, /j, = 5 x 10~^, s = 1. 



•with the cyclic behavior of the system. However, for high 
fi the probability peaks around a point. This point is in- 
terior for the most stochastic rules, but corresponds to 
a defective population for unconditional imitation. The 
simplexes are obtained for the same parameter values as 
used in Fig. [TJ so a direct comparison with the results of 
the replicator dynamics can be made. 



In this paper we have proven that the existence of jok- 
ers, i.e., individuals whose purely destructive behavior is 
directed against the common enterprises represented by 
PG games, allows for the emergence of robust evolution- 
ary cycles in finite populations regardless of the updat- 
ing method chosen. Together with a previous report [l^ 
on the existence of limit cycles for infinite populations 
evolving via a replicator-mutator dynamics, our present 
results show that limit cycles are a generic feature of 
the dynamics generated by destructive agents, not re- 
stricted to a particular selection dynamics. In fact, this 
is a dynamical feature that makes this model different 
from other three-player games like that of loners [l^ , for 
which cycles are structurally unstable and their existence 
strongly depends on the absence of mutations and other 
kinds of perturbations. 

In a recent paper Mobilia has shown that, of the 
three possible outcomes of the replicator equation for 
rock-paper-scissors games [Ij, namely (a) orbits are at- 
tracted towards an asymptotically stable mixed equilib- 
rium, (b) orbits cycle around a neutrally stable mixed 
equilibrium, and (c) orbits go away from an unstable 
mixed equilibrium and approach the heteroclinic orbit 
defined by the border of the simplex (the case of the 
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FIG. 7: (Color online.) Density plots for the probability of finding the system in each population state as obtained by solving 
numerically Eq. First row corresponds to unconditional imitation, second row to proportional imitation, third row to a 
Moran process. In each case mutations increase left to right. In all three cases low mutation rates (fi) yield high probabilities 
near the boundaries of the simplexes, specially near the corners, corresponding to cyclic transitions between homogeneous states. 
Increasing fj, increases the probability to find the system near homogeneous defective populations. For high jj, an attractive 
point appears close to the D corner which goes away from it upon increasing fi. Parameters are n — 5,r — 3,d = 0.4; the 
selection strength is s = 1 in the first and second rows, s = 0.38 in the third. Mutation rates have been chosen as in Fig. [T] and 
appear near each simplex. Densities are plotted using a logarithmic scale. 



9 



Joker game) , adding mutations between the three strate- 
gies merges cases (a) and (b), both of which yielding a 
mixed equilibrium. Oscillations disappear in these two 
cases. The Loners game belongs to class (b). In con- 
trast, the Joker case analysed in this paper belongs to 
class (c), with mutations generating an attractive, stable 
limit cycle. In this case the dynamics oscillates between 
the three strategies with well-defined and robust oscil- 
lations. We are not aware of any other game for which 
the inclusion of a simple behavioral type (jokers do not 
need memory, have no especial recognition abilities, and 
do not rely on any reputation generated along the game) 
leads to cycles which are robust to perturbations and 
have a well defined period and amplitude irrespective of 
the initial fractions of players. 

We have expanded here these results and have proven 
that the oscillatory dynamics does not occur only for in- 
finite (or very large) populations evolving under a repli- 
cator dynamics, but also in the case of finite populations 
and for different update rules. We have analyzed un- 
conditional imitation, proportional update, and a Moran 
process. In all cases the system exhibits finite time lapses 
in which most of the population is composed by cooper- 
ative individuals, finding that the Moran process for low 
(but not extremely low) selection pressures is the most 
favorable to cooperation — as the system spends 50% of 
the time in cooperative states. Under unconditional im- 
itation the system spends one third of the time in co- 
operative states, whereas the more stochastic nature of 
proportional update favors defection due to the slower 
invasion of jokers, and thus the system stays longer in 
defective states — especially so for high mutation rates. 

Let us note that, if the damage d inflicted by jokers is 
zero, jokers are not able to overcome defectors and oscilla- 
tions are supressed. The system ends up in a steady pop- 
ulation where cooperation becomes extinguished, both 
with and without mutations [l3| . Indeed, this case is 
identical to the loner model when the benefit obtained 
by loners is also zero, situation in which both jokers and 
loners become simply non-participants in the game with 
the only effect of reducing the effective number of players 
[l3| . We have shown here that for finite populations 
and 0? = random drift allows for bursts in which the 
system spends some time in fully cooperative states, but 
that the happening probability of such events is very low. 

The existence of damaging agents, which are able to 
destroy the defective populations and lead to a state 
without cooperators and defectors, gives cooperators the 
chance to re-build cooperative enterprises, and thus pro- 
motes cooperation. This result, as well as modifications 
of the model presented here, might be interesting in the 
study of human evolution, where examples of destructive 
periods can be found along history as a result of revolu- 
tions or wars. It has been suggested that these destruc- 
tive periods take place whenever a society reaches a point 



where the public goods fall below a certain threshold [21 



cooperation in animals arises whenever there is a risk or 
they face a predator, but also provide insights into the 
evolutionary cycles observed in human society. 
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Appendix A: Average payoffs in a finite population 

Let us denote Pxi^n, I) the average payoff that a player 
of type X receives when the population is made of m co- 
operators, j jokers, and AI —m—j defectors. This average 
payoff is calculated by averaging the corresponding pay- 
off ([1]) with the probability distribution ([3]) . For defectors 
this implies 

PDim,j)= !:^—^p{k,l\n-l,m,j,M-l). (Al) 



kd>0 
k+Kn 



I 



To perform this average it will prove convenient to fac- 
torize the probability distribution as the product of two 
standard hypergeometric distributions, i.e., 

pikJ\n,mJ,M)=pil\nJ,M)pik\n-l,m,M-j). (A2) 

where 



pil\nj, M) 



l)\n-l 



(A3) 



The first term in (jA2| is the probability of selecting I 
jokers out of the population, and the second term is the 
conditional probability of subsequently selecting k coop- 
erators, given that we have already selected the / jokers. 

A useful identity of the hypergeometric distribu- 
tion — consequence of the properties of the binomial 
coefficients — is 



The Joker game shares this feature. Modifications of the 
model presented here may thus help explain not only how 



kp{k\n, m, M) 



M 



■p(fc- l|n- l,m- 1,A/- 1). (A4) 
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Substituting factorization (jA2p into (|A1[) and making use 
of this identity we readily obtain 



PuimJ) 



n-l , 

rm \ ^ 71 — t — 1 



M - 7 - 1 ^ n - i 

•' 1=0 

n-l , 



pG|n-l,j, A/-1) 



-pil\n-l,j,M-l). 



1=0 



A new identity, namely 
p{l\n-l,j,M -I) _ M 



n — I n(M — j) 

allows us to do the sum 



(A5) 

pil\n,j,M), (A6) 



|4 pil\72^1,j,M-l) ^ M[l-p{n\n,j,M)] ^^^^ 



1=0 



n — I 



n{M - j) 



It will prove convenient to introduce the function 



E{n,j, M) 



J 



1 - 



M-j 
in terms of which 

l-p{n\n,j,M) = l- 



(j - 1) ■ ■ ■ (j -n + 1) 
{M - 1) • • • (A/ - n + 1) 



(A8) 



jXj - 1) • • • (j - n + 1) 



M{M - l)---(A/-n+l) 
M-j 



M 



[1 + S(n,j, Af)] 



This allows us to write 

""'pG|n-l,j,M-l) l+S(n,i,M) 



n — / 



n 



(A9) 
(AID) 



and using this in (jASP obtain 



D / -N rm[n - 1 - ^(n, j, M)] . 



n{M -j - 1) 

As for the average payoff of a cooperator, 
r{k + l) -dl 



(All) 



PcimJ) 



-1+ E 



/c,i>0 
k+Kn 



I 



X p(fc,Z|71 — 1,771 — 1, A/ — 1) 



^E 



pil\n-l,j,M -1) 

n — I 



l + PD(m-l,j) 



-[l + S(77,j,A/)]-l + PD(7n-l,j). 

77 



(A12) 



Therefore 



, r f (n- l)(m - 1)\ 

777—1 



r 

n \ M -j -1 



E{n,j,M). 

(A13) 



Finally, P:j{n,j) = because jokers get zero regardless 
of the composition of the population. 



Appendix B: Calculation of the transition matrices 

Transition probabilities T[m, j\m' , j') are obtained ac- 
cording to the specified update rule. We will calcu- 
late those corresponding to the rules used in this work. 
But before we proceed let us introduce some shorthands. 
We will write T^^^^^ = T{ra,j\m + ei, j -I- £2), where 
El, £2 G {^1, 0, !}• Also by w^^^ we will denote the prob- 
ability that a player of type Y is chosen to be replaced 
by a player of type X when the population is made of 
771 4- £1 cooperators, j + £2 jokers, and 7\/ — 777, — j — £1 — £2 
defectors. Whether the Y player is finally replaced by an 
X one depends on mutations, thus 



7^1.0 




-2/i) + 


{-i'o + 






Ti.-i 


= <-i(l 


-2m)- 








T-1,0 




-2m)- 








To,~i 




-2m)- 








T-iA 




-2m)- 




+ --m) 




To,i 




2m) + 


{^o.i + 







(Bl) 



In all cases there are two possibilities for a Y individual 
to become an X one, either a pair XY is selected, the 
update takes place and no mutation occurs, or another 
pair ZY is selected (with Z 7^ X) but a mutation changes 
Z into X. 

Finally, the probability that no change of strategy oc- 
curs Tq^o = T{m, j\m, j) is obtained as 



To.o = 1 - (1 - m) 



E<o 



2mE' 



XX 

^0,0 ' 



(B2) 



where the subscript 0, refers to a population made of 
771 cooperators, j jokers, and M — m — j defectors. 

Notice that the expansion p7p readily follows from ex- 
pressions (|BT|) and (|B2|l . 



1. Unconditional imitation 

This rule prescribes that two players are selected at 
random from the population and the strategy of the 
model player (X) replaces that of the focal player (Y) 
if the latter has a lower payoff. Accordingly, if X 7^ Y, 



,XY 



"■ei,e2"'ei,e2 o (' nX 



(B3) 



where 8(a;) = 1 if a; > and otherwise, and 71?- ^ de- 
notes the number of individuals of type X in the popula- 
tion (e.g., 77^0 = 771-^1, 7lf_o = AT-j-TTl-l, 71^ _i = j-1. 
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= M — j — m, etc.). On the other hand, in order to 
account for mutations we must define 



XX _ 

M(M-l) ■ 



(B4) 



condition), so under this assumption C always in- 
vades J, but J never invades C. 
(iii) P,7(0,j) > Pd(0, j) for ah < j < M, so J always 
invades D, but D never invades J. 

Therefore 



2. Proportional update 



Similarly to the previous rule. 




(CI) 



X Y 

XY ^ "^i.e^^ei.e, . X _p. 



7>Y 



M{M - 1) 



(B5) 



where 'I'(x) = x/Q, if a; > and otherwise, VI being a 
constant ensuring that ^ (^"£^,£2 ~ ^^1,^2) — ^ (typically 
51 is chosen as the largest possible payoff difference). As 
in the previous rule w^'^^ is given by (jB4p . 



This implies 



ac = au — aj 



(C2) 



On the other hand, ifr < l + (n — l)d neither C invades 
J nor vice-versa, so in this case 



3. Moran process 



In this case payoffs are replaced by fitnesses = 
1 — s + sP]^ (sec Sec, nil D|) . Let us introduce the total 
fitness of the population 



(B6) 



which implies 




ac = «D = 0, aj = 1. 



(C3) 



(C4) 



The Moran rule specifics that a player is chosen for repro- 
duction proportional to its fitness and the offspring re- 
places another randomly chosen individual from the rest 
of the population. So if X ^ Y, 



„Y X pX 

jXY _ "'il,t2 "■ei,e2^<;i,e2 

M-1 ' 



(B7) 



and 



2. Moran Process 

The Moran process for a population with two strategies 
defines a birth-death process with two absorbing states. 
The details of the calculation of pyx can be found in [ll| 
and follow standard formulae for this kind of processes 
[2^ . Summarizing, if we denote Pyxim) the payoff re- 
ceived by a type Y individual when the population is 
made of to Y individuals and M — m X individuals, then 



_ 1 „x pX 

XX _ "'ei,e2 ^ "•ei,£2^<:i,e2 



(B8) 



M-l 



PXY — Im-iPyx, 



(C5) 



Appendix C: Stationary probabilities in the wealc 
mutation limit 



1. Unconditional imitation and proportional 
update 

According to the payoffs obtained in Appendix |XJ 

(i) PD{m,Q) > Pc{m,0) for all < ?7i < M, so D 
always invades C, but C never invades D. 

(ii) Pc{m, M-m) > Pjijn, M-m) for all < to < M, 
provided r > 1 + [n — l)d (the rock-paper-scissors 



where go = 1 and 

I - s + sPxy{M - m) 

q-m^qm-i — ; — -p, — -, — ^ — , \)<m<M. 

1 - s + sPyxim) 

(C6) 

Payoffs Pxyi'm) and Pyxi'm) are obtained from the for- 
mulae of Appendix [Xj The maximum value of the selec- 
tion strength s is given by 



1 



min Pxy(w) 

XY,m 



(C7) 
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